Add Apertus tool parser - #1663

Apertus emits tool calls as an array of single key objects wrapped in
<|tools_prefix|> … <|tools_suffix|>:

<|tools_prefix|>[{"get_weather": {"location": "London"}}]<|tools_suffix|>

This adds a parser for that format and infers it from the chat template, so
Apertus models work without an explicit tool_parser_type in
tokenizer_config.json. Before this, _infer_tool_parser returned None for
Apertus, has_tool_calling stayed False, and the server rejected any request
carrying tools.

It also fixes reasoning detection for the same models. Apertus delimits
deliberation with <|inner_prefix|> … <|inner_suffix|>, but it also carries
unused <think>/</think> tokens in its vocabulary (ids 69/70). Since
_infer_thinking scans the vocab, it matched the unused pair, so the reasoning
state was never reachable and deliberation came back as content instead of
reasoning. The Apertus pair is now checked first.

Both changes are in one PR because they enable the same models and touch the
same file, following #810, which added the LongCat tool parser and its
THINK_TOKENS entry together.

Provenance

Derived from vLLM. parse_tool_call is adapted from vLLM's Apache-2.0
licensed
apertus_tool_parser.py
— specifically its non-streaming extract_tool_calls, whose structure it keeps:
wrapping a non-list payload, skipping entries that aren't non-empty objects, and
next(iter(obj.items())) to split the single key into name and arguments.
Reduced to the batch path, since mlx-lm buffers the whole tool region and parses
it once, so vLLM's streaming diff machinery has no analogue here. The header
follows kimi_k2.py, which is likewise a "Modified from" vLLM port. Flagging the
Apache-2.0 → MIT direction explicitly in case you'd prefer different attribution.

Informed by SGLang, but not derived from it. No code is taken from SGLang's
apertus2509_detector.py;
its equivalent helper is written differently. What it contributed is one
behavioural decision — normalizing null arguments to {}, so clients never
receive "arguments": "null" — and corroboration of two facts: the wire format,
and the reasoning marker names, which match its Apertus2509Detector in
reasoning_parser.py. Both facts were also verified directly against Apertus'
chat template and vocabulary rather than taken on trust.

Deliberately not carried over from either: partial-JSON recovery for calls
truncated mid-generation (vLLM's test_incomplete_tool_call). No mlx-lm parser
does this, and since tool regions are parsed once rather than incrementally, a
truncated call raises and is logged by the server instead.

Note on <|tools_suffix|>

Worth flagging for future reference: Apertus lists <|tools_suffix|> in
eos_token_id ([2, 68, 72]), so its tool-call end marker is also an EOS
token. This works on current main because stop-sequence matching
(StopSequenceMatcher) is separate from the text state machine. It did not
work in 0.31.3, where EOS ids were folded into the tool state's transition trie
as stop edges — the duplicate (72,) overwrote the tool-end transition, so tool
calls were parsed into a buffer and then silently discarded (HTTP 200, empty
assistant message). No change is needed for this, but it's a real constraint on
that part of the design.

Testing

• tests/test_tool_parsing.py::test_apertus — single and multiple calls, nested
  arguments, null arguments, bare object instead of an array, unparseable
  entries skipped, and ValueError on nothing-parseable and on a call truncated
  mid generation.
• tests/test_tokenizers.py::test_thinking_marker_precedence — locks in that
  Apertus' markers win over the unused <think> pair. Uses a stub vocab so it
  needs no network, following test_find_token's precedent of testing internals.
• tests/test_tool_parsing.py, tests/test_server.py (25 tests), and the
  targeted tokenizer tests pass locally. pre-commit (black, isort) clean.
• Verified end to end against mlx_lm.server with
  tokimoa/apertus-v1.5-8b-mlx-8bit: tool_calls emitted with
  finish_reason: "tool_calls" on both the batch and streaming paths, and a
  1032-char reasoning field correctly separated from content with
  enable_thinking: true.
• Template inference confirmed on both swiss-ai/Apertus-v1.5-8B and the
  quantized tokimoa/apertus-v1.5-8b-mlx-8bit, neither of which carries a
  tool_parser_type key.

Not run locally: the full tests/test_tokenizers.py suite, which downloads
several multi-GB models.

Follow-up, deliberately not in this PR

_infer_thinking resolves markers by scanning the vocabulary, which makes it
order-dependent for any model shipping unused marker tokens — Apertus is the
first case, but it won't be the last, and the failure is silent (reasoning
appears as content). Keying off the chat template, as _infer_tool_parser does,
would be more robust. Worth raising as an issue rather than expanding this PR.